feat: add the ai-queue-worker service [ENG-2181] by agoldis · Pull Request #40 · currents-dev/docker

agoldis · 2026-11-10T04:26:42Z

Linear: ENG-2181

Adds the ai-queue-worker service, off by default in an ai profile. On-prem webhooks no longer runs AI tasks after the Currents release that contains currents-dev/currents#4874, so an install that uses AI needs this service turned on.

This must ship with, not before, the Currents release that contains currents-dev/currents#4874. The image ai-queue-worker does not exist before then, so docker compose pull fails for the new service on any earlier tag. Merge it into the release that sets DC_CURRENTS_IMAGE_TAG to that tag.

Changes

templates/compose.currents.yml: service ai-queue-worker, image ${DC_CURRENTS_IMAGE_REPOSITORY}ai-queue-worker:${DC_CURRENTS_IMAGE_TAG}, built like webhooks (env_file: .env, CURRENTS_ENV: onprem, no ports, no command override).
Off by default. The service is in the ai profile, as traefik is in tls, and starts only with COMPOSE_PROFILES=ai in .env. It needs an LLM endpoint the admin provides. The docs don't yet say what AI features an admin gets, how we use their provider, or what it costs (ENG-2209). AI is also off on-prem until an org admin grants consent.
The AI indexes. Turning AI on also takes AI_QUEUE_WORKER_ENABLED=true in .env, so the scheduler builds the AI collections' MongoDB indexes, including the TTLs that expire AI sessions. The director builds indexes only with CREATE_MONGO_INDEX, which no compose file sets.
Redis and MongoDB depends_on, and the RustFS FILE_STORAGE_FORCE_PATH_STYLE, are copied from webhooks in the three data-service templates.
Compose files regenerated with scripts/sync-templates.sh.
.env.example and docs/configuration.md: how to turn the worker on (COMPOSE_PROFILES=ai, AI_QUEUE_WORKER_ENABLED=true), and the LLM_* and AI_QUEUE_CONCURRENCY settings. No new required variable.
docs/container-images.md: ECR policy and both image-mirroring lists. on-prem/README.md: services list.
on-prem/CHANGELOG.md: Unreleased entries with the upgrade notes below.
webhooks and the director are unchanged. No compose file sets AI_QUEUE_WORKER_ENABLED.
No VERSION or image tag change; the release PR does that.

Upgrade notes for on-prem customers

If you use AI today, it stops at this upgrade until you turn the worker on. Add COMPOSE_PROFILES=ai to .env, and keep AI_QUEUE_WORKER_ENABLED=true and your LLM_PROVIDER_URL / LLM_API_KEY.
To turn it on, add the currents/on-prem/ai-queue-worker repository to your ECR policy and mirror the image with the others.
Regenerate your compose file if you use custom templates.
In webhooks, AI_QUEUE_WORKER_ENABLED=true now only starts the Slack AI answer worker, and only when SLACK_AI_CALLBACK_WORKER_ENABLED is unset.

Checked

docker compose config --quiet on all three files with the validate workflow's minimal .env: valid. config --services leaves out ai-queue-worker by default and includes it with COMPOSE_PROFILES=ai; with tls,ai it includes traefik too.
scripts/sync-templates.sh was run after the profile change.
docker compose -f docker-compose.{full,database,cache}.yml config --quiet with a minimal .env, as the validate workflow does: all three pass.
scripts/sync-templates.sh: the committed files match the templates.
Not run: the Podman validation, and the smoke test (it starts only scheduler and api, so it does not start the new service).

Checklist

PR title is Changelog-friendly: (fix|feat|chore): title [CSR-xxx]
User-friendly and meaningful description of the changes
Documentation was updated

🤖 Generated with Claude Code

https://claude.ai/code/session_011baD3izoabUn31Y3WeQHMp

On-prem webhooks no longer runs AI tasks. The new ai-queue-worker service consumes the ai-bmq queue; without it, AI explanations and chat stop. - Service in compose.currents.yml, same image repository and tag as the other services, same .env. Redis and MongoDB dependencies and the RustFS path-style setting are copied from webhooks. Compose files regenerated. - .env.example and the configuration reference list the LLM_* settings. - Docs: services list, ECR policy and image mirroring lists, changelog upgrade notes. The image exists only in the Currents release that contains currents-dev/currents#4874. Release this with that image tag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011baD3izoabUn31Y3WeQHMp

baz-reviewer · 2026-11-10T04:26:50Z

Baz Summary

Add the optional ai-queue-worker service to run on-prem AI tasks from ai-bmq after webhooks stops processing them. Configure its Docker Compose profiles, dependencies, environment settings, image distribution, and upgrade documentation so administrators can enable AI safely with their own LLM endpoint.

Topics

Topic Details

AI Deployment Guidance

Explain the upgrade impact and add the worker image to ECR access and mirroring workflows, including guidance that existing AI users must explicitly enable the new service.

Modified files (3)

docs/configuration.md
docs/container-images.md
on-prem/CHANGELOG.md

Latest Contributors(2)

User	Commit	Date
dj@currents.dev	docs: say how to turn ...	November 12, 2026
agoldis@gmail.com	feat: add the ai-queue...	November 10, 2026

On-Prem AI Worker

Add the profile-gated ai-queue-worker service to all on-prem Compose variants, configure MongoDB, Redis, and RustFS integration, and document enabling it alongside AI_QUEUE_WORKER_ENABLED and LLM settings.

Modified files (9)

on-prem/.env.example
on-prem/README.md
on-prem/docker-compose.cache.yml
on-prem/docker-compose.database.yml
on-prem/docker-compose.full.yml
on-prem/templates/compose.currents.yml
on-prem/templates/compose.mongodb.yml
on-prem/templates/compose.redis.yml
on-prem/templates/compose.rustfs.yml

Latest Contributors(2)

User	Commit	Date
dj@currents.dev	docs: say how to turn ...	November 12, 2026
agoldis@gmail.com	feat: add the ai-queue...	November 10, 2026

Merger ✅ Ready to Merge

MERGE: the new AI queue worker and its enablement/documentation are present, CI ran, and all review discussions are settled.

_{Activate Skill Maintainer to keep your skills up to date}

_{This PR would have been improved with Baz Planner - Try it now}

twk3

Ran this locally on images built from main 6f87424c7 (includes currents-dev/currents#4874), full compose file, with USE_MOCK_AI=true. The worker starts and webhooks no longer takes AI tasks. testSidebarAiAnalysis.spec.ts passed with its GraphQL stub removed: both requests in aiRequests completed, and only ai-queue-worker logged them.

One change requested: put the service behind a Compose profile so it is off by default for now. AI is already off per org until features.aiAnalysis is set, and the worker needs LLM_* settings to do anything, so most installs would run an idle container and have to pull and mirror an image they don't use. With a profile, docker compose pull and up skip it unless COMPOSE_PROFILES includes ai. traefik already does this with tls. Comments inline.

Baz withdrew its approval because this pull request is no longer ready to merge.

The worker needs an LLM endpoint the admin provides, and the docs do not yet say what AI does with it or what it costs (ENG-2209). It now sits in an `ai` profile, as traefik sits in `tls`, and starts only with COMPOSE_PROFILES=ai in .env. The changelog also said the director reads AI_QUEUE_WORKER_ENABLED to create the AI indexes. The director builds indexes only with CREATE_MONGO_INDEX, which no compose file sets; the scheduler builds them, and reads the variable from the same .env. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The TLS instructions run `docker compose --profile tls up`, and a --profile flag replaces COMPOSE_PROFILES. A TLS user who sets COMPOSE_PROFILES=tls,ai and keeps the flag gets Traefik but no ai-queue-worker, and AI requests wait in ai-bmq. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Removing ai from COMPOSE_PROFILES and running docker compose up -d leaves the worker running with its old settings; --remove-orphans does not remove it either. docker compose rm -sf ai-queue-worker does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

twk3 · 2026-11-12T22:45:25Z

Tested docker-compose.full.yml at 82b4068 on Docker Desktop, with the seven release/2026-11-10-003 images built locally. The .env came from scripts/setup.sh --env-only, with the image tag set and the host ports moved.

1. Default .env

All 10 services started, without ai-queue-worker or traefik.
The scheduler has no AI_QUEUE_WORKER_ENABLED. Its startup tasks completed, and aiSessions, aiSessionMessages and aiSlackCallbacks weren't created.
I seeded a run with failed tests, granted the org FAILURE_ANALYSIS consent, and sent POST /v1/ai-requests (errorExplanation). The request stayed pending after 20 s, with its job in bull:ai-bmq:wait.

2. Worker on, following .env.example

I uncommented COMPOSE_PROFILES=ai and AI_QUEUE_WORKER_ENABLED=true, set LLM_PROVIDER_URL, LLM_API_KEY and USE_MOCK_AI=true, and ran docker compose up -d.

ai-queue-worker started with 0 restarts. The scheduler finished its startup tasks 19 s after up.
The pending request went to completed with success: true, and the queue emptied. Worker log: "Processing test error analysis", "AI generator completed", "test error analysis completed".
A new testExplanation request completed in 44 ms.
The scheduler built the indexes on the three AI collections, including the expiresAt_1 TTL indexes.
Every app service was recreated, not only the worker, because they all read .env.

3. TLS

With COMPOSE_PROFILES=tls,ai:

docker compose up -d starts traefik and ai-queue-worker.
After docker compose down, docker compose --profile tls up -d starts traefik only. That's the case Baz flagged, which 82b4068 fixes in the docs.

4. Turning it off

With ai removed from COMPOSE_PROFILES, docker compose up -d left ai-queue-worker running with its old settings. --remove-orphans didn't remove it either. docker compose rm -sf ai-queue-worker did.
dcf7ac6 adds that to docs/configuration.md and .env.example.

Not covered:

A real LLM provider. USE_MOCK_AI returns a fixed answer without calling one.
Pulling the images from ECR.
MongoDB in the last step. After the down, MongoDB couldn't reopen its files in the macOS bind-mounted data/mongodb ("Operation not permitted"), so the services in step 4 ran without a database. I've seen the same on this Docker Desktop VM before, outside this PR. It doesn't change the step 4 result, which is only about which containers run.

Baz withdrew its approval because this pull request is no longer ready to merge.

agoldis requested a review from twk3 November 10, 2026 04:27

agoldis assigned twk3 Nov 10, 2026

baz-reviewer Bot added the baz: ready to merge label Nov 10, 2026

baz-app Bot previously approved these changes Nov 10, 2026

View reviewed changes

twk3 reviewed Nov 10, 2026

View reviewed changes

Comment thread on-prem/templates/compose.currents.yml Resolved

Comment thread on-prem/.env.example Resolved

Comment thread on-prem/CHANGELOG.md Resolved

baz-reviewer Bot added baz: needs human review and removed baz: ready to merge labels Nov 10, 2026

twk3 requested changes Nov 10, 2026

View reviewed changes

twk3 mentioned this pull request Nov 12, 2026

docs: list the 2026-11-10 index builds and migrations with measured times [ENG-2184] #41

Draft

3 tasks done

baz-reviewer Bot reviewed Nov 12, 2026

View reviewed changes

Comment thread docs/configuration.md Resolved

baz-reviewer Bot removed the baz: needs human review label Nov 12, 2026

baz-app Bot previously approved these changes Nov 12, 2026

View reviewed changes

baz-reviewer Bot added the baz: ready to merge label Nov 12, 2026

baz-reviewer Bot removed the baz: ready to merge label Nov 12, 2026

baz-reviewer Bot added the baz: ready to merge label Nov 12, 2026

baz-app Bot approved these changes Nov 12, 2026

View reviewed changes

twk3 approved these changes Nov 12, 2026

View reviewed changes

twk3 merged commit 8021304 into main Nov 12, 2026
5 checks passed

twk3 deleted the agoldis/eng-2181-ai-queue-worker branch November 12, 2026 22:48

Navigation Menu

Repository navigation

feat: add the ai-queue-worker service [ENG-2181] - #40

feat: add the ai-queue-worker service [ENG-2181]#40
twk3 merged 4 commits into
mainfrom
agoldis/eng-2181-ai-queue-worker

agoldis commented Nov 10, 2026 •

edited by twk3

Loading

Uh oh!

baz-reviewer Bot commented Nov 10, 2026 •

edited

Loading

Baz Summary

Topics

Merger ✅ Ready to Merge

Uh oh!

twk3 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

twk3 commented Nov 12, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

agoldis commented Nov 10, 2026 • edited by twk3 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Upgrade notes for on-prem customers

Checked

Checklist

Uh oh!

baz-reviewer Bot commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Baz Summary

Topics

Merger ✅ Ready to Merge

Uh oh!

twk3 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

twk3 commented Nov 12, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

agoldis commented Nov 10, 2026 •

edited by twk3

Loading

baz-reviewer Bot commented Nov 10, 2026 •

edited

Loading